Deprecate non-ClickHouse / non-Postgres destination connectors by Jeremyyang920 · Pull Request #4491 · PeerDB-io/peerdb

Jeremyyang920 · 2026-06-24T17:15:32Z

Implements the code/config deliverables for #4465 — signaling the deprecation of the non-ClickHouse / non-Postgres destination connectors. No connector code is removed; deprecated connectors stay fully functional.

Deprecated destinations: Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent + Redpanda), Azure Event Hubs, Google Pub/Sub, and S3. Actively-maintained paths: Postgres → ClickHouse / ClickHouse Cloud / Postgres.

Changes

README — deprecation notice placed just after the intro (above the fold) + a connector status matrix. BigQuery is shown as a maintained source and deprecated destination.
docs/deprecated-connectors.md (new) — pin + fork migration guide, per-connector paths.
CONTRIBUTING.md (new) — links to the migration guide.
UI (peer-types route + SelectSource) — uniform "Deprecated" badge in the peer-creation grid. Since peers are direction-agnostic and BigQuery is a valid source, BigQuery is marked "Deprecated*" with a footnote: "Deprecated as a destination only."
GitHub automation — issue form with a destination-connector dropdown + deprecation notice; .github/labeler.yml + deprecated-connector.yml workflow that labels deprecated-connector PRs and posts a single de-duplicated notice (never auto-closes).

Notes / follow-ups (not in this PR)

The deprecated label must be created once in repo Settings → Labels (labels aren't defined in-repo).
Post-merge: apply the deprecated label + standard comment to existing open non-CH destination issues/PRs (do not auto-close).
BigQuery is deprecated only as a destination — it remains a supported source.

Verification

Brought the full dev stack up (after generate-protos.sh); verified /api/peer-types returns the correct per-connector deprecated/deprecatedRole data and /peers/create renders the badges + footnote without errors.
All issue-form / labeler / workflow YAML validated.
GitHub automation validated end-to-end on a fork (see the pinned summary comment).

🤖 Generated with Claude Code

github-actions · 2026-06-24T18:20:19Z

🔄 Flaky Test Detected

Analysis: The deterministic unit test TestRunPipeline_FilterStripsLines failed with "src failed: signal: killed" — its trivial printf subprocess was SIGKILLed (OOM/resource pressure) under the heavily parallel CI run, not due to a code bug.
Confidence: 0.88

✅ Automatically retrying the workflow

View workflow run

Jeremyyang920 · 2026-06-24T21:33:19Z

Tested the GitHub automation end-to-end on a fork (Actions enabled, branch as default, deprecated label created). This surfaced a real bug in the labeler config from my previous commit:

The all: + all-globs-to-all-files negation block silently disabled the entire actions/labeler@v5 config — the label job succeeded but applied no labels at all (even a trivial flow/connectors/snowflake/** match was skipped). No error is logged; the action just no-ops.

Fix (d78c1d7b): dropped the all: block and instead list BigQuery's destination-specific files explicitly, so BigQuery source changes (source.go, source_schema.go, qrep_object_pull.go, bigquery_source_test.go) are still never auto-labeled.

Verified on the fork after the fix:

Scenario	Result
PR touching `flow/connectors/snowflake/**`	✅ `deprecated` label + notice comment
PR touching only BigQuery source files	✅ not labeled, no comment
PR touching BigQuery destination (`bigquery.go`)	✅ `deprecated` label + notice comment
Manual `deprecated` label on an issue	✅ notice comment posted

The needs: label + if: always() change on the notice job also confirmed working: on pull_request_target the notice posts after the labeler applies the label.

codecov · 2026-06-24T21:51:37Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2919	2	2917	338

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot

Stack Traces | 5.23s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot
2026/07/06 15:22:08 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/06 15:22:08 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/06 15:22:08 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_zse7migc.test_mysql_ghost_schema
    clickhouse_mysql_test.go:1971: WaitFor waiting on initial 2026-07-06 15:22:13.739667231 +0000 UTC m=+871.966058476
2026/07/06 15:22:13 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_xbxbydfv.test_default_partition_key_parallel
    clickhouse_mysql_test.go:1983: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:1983
        	Error:      	Received unexpected error:
        	            	can't scan into dest[0] (col: rows_in_partition): cannot scan NULL into *int64
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Default_Partition_Key_Parallel_Snapshot (5.23s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-06-24T21:55:57Z

🔄 Flaky Test Detected

Analysis: The lone failure, TestRunPipeline_FilterStripsLines, died with "src failed: signal: killed" (empty stderr) on a trivial in-memory printf→cat pipeline that touches no DB/network — a classic OOM/resource-exhaustion kill under -p 32 high concurrency, unrelated to the PR's destination-connector deprecation change.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-06-24T21:59:32Z

🔄 Flaky Test Detected

Analysis: Test_MySQL_BinlogIncident failed on a transient MySQL handshake EOF ("connection was bad") from its ephemeral debug testcontainer — a network/connection flake unrelated to this destination-connector-deprecation PR.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

Jeremyyang920 · 2026-06-24T22:07:55Z

✅ Current scope + validation (supersedes earlier notes)

Posting as a comment rather than editing the PR description, per this repo's CLAUDE.md policy ("never modify PR descriptions/titles; report changes as PR comments"). The description's note that "S3 is intentionally not deprecated" is now out of date — see below.

Deprecated destination connectors (final scope)

Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent/Redpanda), Azure Event Hubs, Google Pub/Sub, and S3.

S3 is now deprecated (destination-only; it isn't a source). README matrix, docs, the UI badge, the labeler, and the issue-form dropdown/notice all include it.
BigQuery is deprecated only as a destination — it remains a supported source (badge reads "Deprecated*" with a footnote; labeler excludes BigQuery source files).
ClickHouse / ClickHouse Cloud / Postgres (and Postgres/MySQL/MongoDB sources) remain actively maintained.

Validated end-to-end on a fork (Jeremyyang920/peerdb)

Actions enabled, branch set as default, deprecated label created.

Path-based PR labeling (actions/labeler)

PR #2 — flow/connectors/snowflake/** → labeled + notice ✅
PR #3 — only BigQuery source (source.go) → not labeled ✅
PR #4 — BigQuery destination (bigquery.go) → labeled + notice ✅

Issue-form dropdown labeling

Issue #5 — selects Snowflake → labeled + notice ✅
Issue #6 — selects ClickHouse (maintained) → not labeled ✅
Issue #7 — selects BigQuery (destination) → labeled + notice ✅

Manual labeling

Issue #1 — maintainer adds deprecated → notice posted ✅

Notice comments are de-duplicated (hidden marker) and nothing is ever auto-closed. (S3 follows the same destination-only mechanism already proven for Snowflake/ElasticSearch above.)

One-time setup before this works upstream

Create the deprecated label in PeerDB-io/peerdb → Settings → Labels.

Jeremyyang920 · 2026-06-24T22:43:41Z

Thanks @ilidemi — addressed in b8fbdbe:

labeler.yml header — now lists S3 as deprecated and notes ClickHouse/Postgres as the excluded (actively-maintained) destinations.
labeler.yml (line ~47) — applied your suggestion: added flow/connectors/bigquery/qrep.go to the tracked BigQuery destination files.
CONTRIBUTING.md — added S3 to the deprecated list (and noted BigQuery is deprecated only as a destination).
bug_report.yml — removed the duplicate migration-guide link.
config.yml / feature requests — good catch; added a feature_request.yml template so feature requests have a proper home while keeping the chooser curated (blank issues stay off).
README → linked status doc — that's the external docs.peerdb.io supported-connectors page; it'll be updated separately as part of this effort, so no in-repo change here.

claude · 2026-06-24T22:49:26Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-06-24T23:49:08Z

🔄 Flaky Test Detected

Analysis: The lone failing test TestRunPipeline_FilterStripsLines died with "src failed: signal: killed" — its trivial printf subprocess was externally SIGKILLed (OOM/resource contention on the heavily-loaded -p 32 runner), which is an environmental flake rather than a logic bug.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-06T15:41:01Z

🔄 Flaky Test Detected

Analysis: The test reads qrep_partitions.rows_in_partition right after data lands in ClickHouse, but that column is populated by a pull goroutine that races the concurrent sync goroutine, so it can still be NULL — confirmed intermittent since the identical query passed in sibling tests and in the same test on other matrix jobs.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

Signal deprecation of the Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent/Redpanda), Azure Event Hubs, Google Pub/Sub, and S3 *destination* connectors. No code is removed; they remain fully functional. Actively-maintained paths: Postgres -> ClickHouse / ClickHouse Cloud / Postgres. - README: deprecation notice (above the fold) + connector status matrix; BigQuery is listed as a maintained source and a deprecated destination. - docs/deprecated-connectors.md: pin + fork migration guide. - CONTRIBUTING.md: links to the migration guide. - UI: "Deprecated" badge in the peer-creation grid. BigQuery is dual-role, so it is marked destination-only ("Deprecated*" + footnote) since it is still a valid source peer. Refs #4465 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Issue chooser offers only the Bug report form, which includes a destination-connector dropdown and a deprecation notice. - labeler.yml: label PRs touching deprecated destination connectors. List BigQuery destination files explicitly so BigQuery *source* changes are never auto-labeled (a `!`-negation / all-globs-to-all-files block was found to silently disable the whole actions/labeler v5 config). - workflow: apply `deprecated` to bug reports that select a deprecated destination, label deprecated-connector PRs, and post a single de-duplicated notice. It never auto-closes. The `deprecated` label must be created once in repository settings. Refs #4465 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- labeler.yml: header comment lists S3 as deprecated and notes ClickHouse/Postgres as the excluded (maintained) destinations; add the BigQuery destination QRep file (qrep.go) to the tracked paths. - CONTRIBUTING.md: add S3 to the deprecated list; note BigQuery is deprecated only as a destination. - bug_report.yml: drop the duplicate migration-guide link. - feature_request.yml: new issue-form template so feature requests have a home while the chooser stays curated (blank issues remain disabled). - Issue forms: add an optional "Source connector" dropdown and make the destination dropdown optional, so source-side issues aren't forced to pick a destination (GitHub issue forms have no conditional fields). The destination dropdown keeps its exact label so the deprecation-triage workflow still parses it. Refs #4465 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Mark the Query Replication (QRep) and XMin mirror types as deprecated; CDC is the recommended, actively-maintained mirror type. They remain fully functional and no code is currently being removed. - ui/app/api/mirror-types: flag QRep and XMin cards as deprecated. - ui/app/mirrors/create/mirrorcards: render a "Deprecated" badge on those cards in the create-mirror flow. - README: add a mirror-type deprecation notice; soften "no code is being removed" to "currently" across the connector/mirror notices. - CONTRIBUTING.md, docs/deprecated-connectors.md: same "currently" wording. Refs #4465 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Branch was behind main's prettier 3.8.3 / TypeScript 6.0.3 lockfile bump (#4536), causing a stale Prettier check failure. Rebasing picks up the updated formatting; this empty commit re-runs CI against the current tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…or types (#246) This documentation PR accompanies PeerDB-io/peerdb#4491. It adds concise deprecation callouts (no content removed or restructured) to mark the following as deprecated and no longer actively maintained. They remain fully functional and no code is currently being removed. ## Deprecated destination connectors - Snowflake - ElasticSearch - Kafka (including the Confluent and Redpanda variants) - Azure Event Hubs - Google Pub/Sub - S3 - GCS - BigQuery — deprecated **only as a destination**. BigQuery remains a **supported source**. Actively-maintained destinations: ClickHouse, ClickHouse Cloud, Postgres. ## Deprecated mirror types - Query Replication (QRep) - XMIN CDC (Change Data Capture) is the recommended, actively-maintained mirror type. ## Pages updated - Connector setup guides: `connect/snowflake`, `connect/bigquery`, `connect/s3`, `connect/gcs`, `connect/elasticsearch`, `connect/kafka`, `connect/confluent-cloud`, `connect/pubsub` - CDC mirror guides to deprecated destinations: `mirror/cdc-pg-sf`, `mirror/cdc-pg-bq`, `mirror/cdc-pg-kafka`, `mirror/cdc-pg-elasticsearch` - CDC use-case pages to deprecated destinations: `usecases/real-time-cdc/postgres-to-snowflake`, `postgres-to-bigquery`, `postgres-to-cloud` (S3/GCS), `postgres-to-azure-eventhubs` - Streaming Query Replication (QRep): `usecases/streaming-query-replication/overview` and all sub-pages, plus `tutorials/realtime-streaming-of-query-results` - SQL command reference: `sql/commands/create-mirror` (Streaming Query Results and XMIN sections) - Status/matrix pages: `features/supported-connectors`, `sql/commands/supported-connectors`, `features/feature-matrix` (rows marked _(deprecated)_ and a summary callout added) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Jeremyyang920 requested a review from a team as a code owner June 24, 2026 17:15

Jeremyyang920 force-pushed the deprecate-non-ch-pg-destinations branch from 58a79ad to 8c02950 Compare June 24, 2026 21:50

Jeremyyang920 force-pushed the deprecate-non-ch-pg-destinations branch from 8c02950 to ec0bc91 Compare June 24, 2026 22:03

PeerDB-io deleted a comment from claude Bot Jun 24, 2026

ilidemi approved these changes Jun 24, 2026

View reviewed changes

Comment thread README.md

Comment thread .github/labeler.yml Outdated

Comment thread CONTRIBUTING.md Outdated

Comment thread .github/ISSUE_TEMPLATE/bug_report.yml Outdated

Comment thread .github/ISSUE_TEMPLATE/config.yml

Comment thread .github/labeler.yml

Jeremyyang920 force-pushed the deprecate-non-ch-pg-destinations branch from 41f25b4 to 1287d2a Compare June 24, 2026 22:58

ilidemi approved these changes Jun 25, 2026

View reviewed changes

Jeremyyang920 mentioned this pull request Jun 29, 2026

Deprecate non-ClickHouse/non-Postgres destinations and QRep/XMIN mirror types PeerDB-io/docs#246

Merged

Jeremyyang920 and others added 5 commits July 6, 2026 11:07

Jeremyyang920 force-pushed the deprecate-non-ch-pg-destinations branch from e9e1da9 to c402b9c Compare July 6, 2026 18:07

Jeremyyang920 merged commit 477439e into main Jul 6, 2026
18 checks passed

Jeremyyang920 deleted the deprecate-non-ch-pg-destinations branch July 6, 2026 18:34

Uh oh!

Conversation

Jeremyyang920 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Notes / follow-ups (not in this PR)

Verification

Uh oh!

github-actions Bot commented Jun 24, 2026

🔄 Flaky Test Detected

Uh oh!

Jeremyyang920 commented Jun 24, 2026

Uh oh!

codecov Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Jun 24, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jun 24, 2026

🔄 Flaky Test Detected

Uh oh!

Jeremyyang920 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Current scope + validation (supersedes earlier notes)

Deprecated destination connectors (final scope)

Validated end-to-end on a fork (Jeremyyang920/peerdb)

One-time setup before this works upstream

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jeremyyang920 commented Jun 24, 2026

Uh oh!

claude Bot commented Jun 24, 2026

Code review

Uh oh!

github-actions Bot commented Jun 24, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jeremyyang920 commented Jun 24, 2026 •

edited

Loading

codecov Bot commented Jun 24, 2026 •

edited

Loading

Jeremyyang920 commented Jun 24, 2026 •

edited

Loading